Skip to content

fix: extend zero_division parameter to percentage and range-based metrics#3122

Open
mahimn01 wants to merge 1 commit into
unit8co:masterfrom
mahimn01:fix/metrics-zero-division-pct
Open

fix: extend zero_division parameter to percentage and range-based metrics#3122
mahimn01 wants to merge 1 commit into
unit8co:masterfrom
mahimn01:fix/metrics-zero-division-pct

Conversation

@mahimn01

@mahimn01 mahimn01 commented May 26, 2026

Copy link
Copy Markdown

Checklist before merging this PR:

  • Mentioned all issues that this PR fixes or addresses.
  • Summarized the updates under Summary.
  • Added an entry under Unreleased in the changelog.

Closes #3132.

Summary

Adds consistent zero_division handling to ape, mape, sape, smape, wmape, ope, arre, marre, and coefficient_of_variation.

  • "warn" (default): 0/0 returns 0.0, while non-zero/0 returns np.nan, and logs a warning.
  • "raise": raises a ValueError.

These metrics reuse _safe_scaled_divide with a configurable zero fill. Scaled metrics keep their existing near-zero behavior; percentage and range metrics only treat exact zeros as zero.

Also:

  • ope now accepts a negative actual_series sum.
  • arre and marre handle zero-range components element-wise.
  • Missing-only wmape and ope inputs remain np.nan.

Tests

  • uv run pytest darts/tests/metrics/test_metrics.py — 478 passed
  • Python 3.12 core suite — 8,417 passed, 323 skipped
  • uv run pre-commit run --all-files — passed
  • uv run make --directory ./docs build-all-docs — passed

@mahimn01
mahimn01 requested a review from dennisbader as a code owner May 26, 2026 15:23
@mahimn01
mahimn01 force-pushed the fix/metrics-zero-division-pct branch from 3565785 to d55581a Compare May 31, 2026 01:04
@mahimn01

mahimn01 commented Jun 8, 2026

Copy link
Copy Markdown
Author

Hi @dennisbader — gentle nudge on this when you get a chance. CI hasn't triggered yet (looks like it needs a maintainer approve-and-run for first-time contributors). It extends the #3059 zero_division pattern to the percentage/range metrics (wmape, ope, arre, marre, coefficient_of_variation), with tests. Thanks!

@codecov

codecov Bot commented Jun 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.48%. Comparing base (b0ccd3a) to head (d55581a).
⚠️ Report is 8 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3122      +/-   ##
==========================================
- Coverage   96.55%   96.48%   -0.07%     
==========================================
  Files         160      160              
  Lines       17285    17293       +8     
==========================================
- Hits        16689    16686       -3     
- Misses        596      607      +11     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jakubchlapek

Copy link
Copy Markdown
Collaborator

Hey @mahimn01, thanks for this PR and contributing :). Will review it shortly as this is something we should definitely add, but it would be also great if you could create an issue for this first. Thanks!

@jakubchlapek jakubchlapek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @mahimn01, I've taken an initial look at the code, it looks fine, thanks. The ope clarification makes sense to me. One thing that we would definitely need to add is analogous support to all the percentage metrics (e.g. ape, mape, sape, smape, wmape, ope, arre, marre).
Before reviewing further I would like to discuss the fill semantics. For the scaled metrics the user could pass in the insample metric leading to the zero_division issue in the first place. Here as we don't supply that, there is no issue of matching the periodic series with forecasts. To align this to the existing solution I think what would make more sense than returning nans is returning a perfect forecast (e.g. 0/0 for MAPE returns 0% error). This would then depend on the metric and best score, so we can think on how to structure this best (maybe collapse it with the _safe_scaled_divide as the logic will be quite similar I think.) Let me know what you think on this :)

Comment thread darts/metrics/metrics.py
Comment thread darts/tests/metrics/test_metrics.py Outdated
@mahimn01

mahimn01 commented Jun 9, 2026

Copy link
Copy Markdown
Author

Thanks @jakubchlapek, this makes sense — best-score for 0/0 is the right call, and it actually fixes a latent bug I'd missed: with NaN-everywhere, a genuinely perfect forecast on a degenerate series scores NaN (ope on a zero-sum series with an identical prediction; arre/marre on a constant series predicted perfectly).

I'll collapse _safe_pct_divide into _safe_scaled_divide — one helper with a configurable 0/0 fill (1.0 scaled, 0.0 percentage), x/0 → NaN, and "raise" preserved (keeping each caller's existing message). A nice side effect of the shared helper's element-wise fill: for arre/marre it fixes a case the current code gets wrong — a constant-actual column is currently all-NaN even at timesteps where the per-step error is 0; element-wise gives 0/0 → 0 and x/0 → NaN within the same column.

On scope (full breakdown in the issue): wmape/ope/arre/marre/coefficient_of_variation are already in this PR — I'll switch them to the unified helper. Good news on the rest of the family: sape/smape already encode 0/0 → 0.0 inline and have no x/0 case, so the only delta there is exposing the explicit zero_division knob for the "raise" option. ape/mape are the trickier ones — they still hard-raise on any y_t == 0, and because their denominator is the per-timestep actual (not a column scalar) they need element-wise handling; relaxing MAPE's default is also a larger behavioral change. Would you prefer ape/mape in this PR (default kept as raise, opt into fill via zero_division) or as a focused follow-up?

One small flag: coefficient_of_variation is RMSE/mean (not abs-wrapped, and the mean can be negative), so a zero mean is a legitimate non-degenerate case — I've proposed 0/0 → 0.0 / x/0 → NaN for it but happy to adjust.

Inline replies:

  • (arre, "why removed?") that was the old if not (y_max > y_min).all(): raise(...) guard — the zero-range check moved into the helper, gated behind zero_division (default fills; "raise" re-raises, preserving the legacy behavior). The old message also said "MARRE" inside arre(), so that copy-paste slip drops out.
  • (test) agreed — I'll fold the negative-sum check into test_ope() with a small negative-series helper. I'll also add a 0/0 best-score case to the zero-division test (the current fixtures are all x/0, so they stay NaN and need no change).

I've opened #3132 to track the design. Thanks for the thoughtful review!

@mahimn01

Copy link
Copy Markdown
Author

Hey @jakubchlapek — no rush, just a bump on this one whenever you get a chance.

I think we're aligned on the core: fold the percentage path into the unified _safe_scaled_divide with best-score for 0/0 and x/0 → NaN. I opened #3132 to track the design.

Two calls are yours before I finish the rework:

  1. ape/mape — happy to fold them into this PR or split them into a focused follow-up. They're a bit trickier than the rest since their denominator is per-timestep (so element-wise handling), and relaxing MAPE's default is a slightly bigger behavioural change — so I'd rather defer to you on scope.
  2. coefficient_of_variation's 0/0 fill — I'm leaning 0.0 (RMSE = 0 is a genuinely perfect forecast), flagging it since its denominator can go negative.

Point me either way and I'll turn it around quickly, and rebase at the same time since master's moved.

@jakubchlapek

jakubchlapek commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

Hey @mahimn01, thanks for the update.
Regarding the cv 0/0 fill it definitely makes sense for me to use 0.0 Maybe we could extend the _safe_scaled_divide to accept a best_score/zero_fill/fitting_name parameter. to pass for a metric what should be the output in case of 0/0.

Regarding the ape/mape I would definitely include them in this PR. This being said I am not quite sure what exactly the issue with them is for this approach. Similarly to other metrics (like coefficient_of_variation which you already implemented) ape exposes y_true and y_pred which we would just need to similarly wrap with the division helper. Similarly I don't get what you mean with the relaxing MAPE's default. Could you double check this and clarify? Thanks :)

@mahimn01

Copy link
Copy Markdown
Author

Hey @jakubchlapek, agreed on both.

On ape/mape, you're right. Once _safe_pct_divide is folded into _safe_scaled_divide (which fills element-wise) it falls out with no special handling. You pass errors = y_true - y_pred and scale = y_true, both (t, c), and per element 0/0 gives 0.0 for a perfect forecast and x/0 gives NaN, same as cv. I'll include them.

On MAPE's default, ape and mape currently raise on a zero actual (it's documented), so defaulting zero_division="warn" changes that to warn-and-fill. Same raise→warn shift the PR already makes for the other five, so it stays under the 🔴 note. I'll default them to "warn" and update the docstrings.

I'll unify _safe_pct_divide into _safe_scaled_divide with a best_score arg (1.0 scaled, 0.0 percentage), route wmape, ope, arre, marre, cv, ape and mape through it, and rebase since master moved.

@mahimn01
mahimn01 force-pushed the fix/metrics-zero-division-pct branch 3 times, most recently from 7e13644 to c72b6bb Compare June 28, 2026 00:26
@mahimn01

Copy link
Copy Markdown
Author

Reworked and pushed, also rebased onto master. _safe_pct_divide is now folded into _safe_scaled_divide via the best_score arg (1.0 scaled, 0.0 percentage), and ape, mape, wmape, ope, arre, marre and cv all route through it. 0/0 gives the perfect-forecast score and x/0 gives NaN, filled element-wise so a constant actual_series no longer NaNs a whole component.

One heads-up, since mape is the default metric for backtest and gridsearch, the raise to warn change makes those surface NaN instead of raising on a zero actual, consistent with the #3059 scaled metrics.

@mahimn01
mahimn01 force-pushed the fix/metrics-zero-division-pct branch from c72b6bb to 332f203 Compare June 29, 2026 18:03
Add zero_division to ape, mape, sape, smape, wmape, ope, arre, marre, and coefficient_of_variation.

Share denominator handling with scaled metrics while preserving their near-zero behavior. Percentage and range metrics use exact zeros: 0/0 returns 0.0, nonzero/0 returns NaN, and raise keeps explicit failure behavior.

Handle zero ranges element-wise, allow negative OPE sums, and keep missing-only WMAPE and OPE inputs as NaN.
@mahimn01
mahimn01 force-pushed the fix/metrics-zero-division-pct branch from 332f203 to 0f6cddd Compare July 14, 2026 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Consistent zero-denominator handling for percentage and range metrics

2 participants